You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by "Herring, Adam via users" <us...@mina.apache.org> on 2025/12/16 14:18:40 UTC
RE: Possible second race condition after EPoll spinning causing ClosedSelectorException
Hi Emmanuel,
Thanks for putting in a fix for this in 2.2.5. We are migrating the head version of our software onto that.
We’ve also implemented a workaround on some of our older code lines that are still live, do you think this fix would eventually be applied to 2.0.X though or is that not likely to happen?
Thanks again,
Adam
From: Emmanuel Lecharny <el...@gmail.com>
Sent: 22 October 2025 09:00
To: Herring, Adam <Ad...@broadridge.com>; users@mina.apache.org
Subject: Re: Possible second race condition after EPoll spinning causing ClosedSelectorException
Hi Adam, On 9/25/25 17: 53, Herring, Adam wrote: Hi Emmanuel, Thanks for your helpful response and the link to the original JDK ticket. I guess it was closed without being truly fixed! Re-registering the channel on the new selector is exactly
ZjQcmQRYFpfptBannerStart
This Message Is From an Unknown Sender
You have not previously corresponded with this sender. Use caution when clicking links or opening attachments. Report suspicious emails by clicking the "Report" button in your Outlook toolbar or forward to SIRT@Broadridge.com<ma...@Broadridge.com>.
ZjQcmQRYFpfptBannerEnd
Hi Adam,
On 9/25/25 17:53, Herring, Adam wrote:
Hi Emmanuel,
Thanks for your helpful response and the link to the original JDK ticket. I guess it was closed without being truly fixed!
Re-registering the channel on the new selector is exactly what NioProcessor.registerNewSelector() looks to be doing.
Indeed. The idea is to loop on all the SelectionKey instances, and register each one of them on the new selector.
Looking at this code in the mina sources again while reading your response, I noticed that this method is taking securing selectorLock.writeLock().lock() before it creates and registers the new selector (and releasing it at the end).
yes, except for the allSessionsCount() method which *should* be protected:
protected int allSessionsCount()
{
return selector.keys().size();
}
should be:
protected int allSessionsCount()
{
selectorLock.readLock().lock();
try {
return selector.keys().size();
} finally {
selectorLock.readLock().unlock();
}
}
NioProcessor.setInterestedInWrite(…) though is not getting the lock before it called key.interestOps(newInterestOps) which is where the ClosedSelectorException occurs.
Looking at the SelectionKeyImpl.java code, here is what I see:
we call:
@Override
protected void setInterestedInWrite(NioSession session, boolean isInterested) throws Exception {
SelectionKey key = session.getSelectionKey();
...
key.interestOps(newInterestOps);
}
which translates to:
public SelectionKey interestOps(int ops) {
ensureValid();
return nioInterestOps(ops); <------
}
which calls:
public SelectionKey nioInterestOps(int ops) {
if ((ops & ~channel().validOps()) != 0)
throw new IllegalArgumentException();
channel.translateAndSetInterestOps(ops, this); <---------
interestOps = ops;
return this;
}
which itself calls:
public void translateAndSetInterestOps(int ops, SelectionKeyImpl sk) {
int newOps = 0;
if ((ops & SelectionKey.OP_READ) != 0)
newOps |= PollArrayWrapper.POLLIN;
if ((ops & SelectionKey.OP_WRITE) != 0)
newOps |= PollArrayWrapper.POLLOUT;
if ((ops & SelectionKey.OP_CONNECT) != 0)
newOps |= PollArrayWrapper.POLLCONN;
sk.selector.putEventOps(sk, newOps);
The last line leverage the selector, so yes, we most certainly need to protect the selector inside setInterestedInWrite and setInterestedInRead.
Is this failure to get the lock here the issue?
Thanks,
Adam
From: Emmanuel Lecharny <el...@gmail.com>
Sent: 23 September 2025 13:29
To: Herring, Adam <Ad...@broadridge.com.INVALID>; users@mina.apache.org<ma...@mina.apache.org>
Subject: Re: Possible second race condition after EPoll spinning causing ClosedSelectorException
Hi Adam, comments inline. . . On 9/23/25 10: 46, Herring, Adam wrote: > Hi Apache MINA team, > > I'm seeking advice on how to handle an apparent race condition encountered using mina 2. 0. 27, whether my analysis is correct and if this is
ZjQcmQRYFpfptBannerStart
This Message Is From an Unknown Sender
You have not previously corresponded with this sender. Use caution when clicking links or opening attachments. Report suspicious emails by clicking the "Report" button in your Outlook toolbar or forward to SIRT@Broadridge.com<ma...@Broadridge.com>.
ZjQcmQRYFpfptBannerEnd
Hi Adam,
comments inline...
On 9/23/25 10:46, Herring, Adam wrote:
Hi Apache MINA team,
I'm seeking advice on how to handle an apparent race condition encountered using mina 2.0.27, whether my analysis is correct and if this is a known issue (I haven't identified one open in https://urldefense.com/v3/__https://issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$<https://urldefense.com/v3/__https:/issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$>).
Issue Description
Our application server uses MINA for NIO between a data cache and several client applications. Recently we have seen increased incidence of the following warning-level message in the data cache logs:
Create a new selector. Selected is 0, delta = 0
I found this is logged from the AbstractPollingIoProcessor and is accompanied by a comment block explaining that there is an Epoll race condition that can cause file descriptors not to be considered available and the selector must be closed and a new one registered to prevent 100% CPU use. It's not entirely clear whether this issue is in MINA, the JDK or the operating system, but either way this all seems fine and most of our application instances do not encounter a further problem.
It's a JDK problem, I do'nt think it was ever fixed:
https://urldefense.com/v3/__https://bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$<https://urldefense.com/v3/__https:/bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$><https://urldefense.com/v3/__https:/bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$><https://urldefense.com/v3/__https:/bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$>
On one production instance of our application it looks like a second race condition occurs when the selector is closed and replaced. The greater the frequency of the "Create a new selector. Selected is 0, delta = 0" message, the more likely that we encounter a ClosedSelectorException on another thread. Full stack trace and logs before and after:
WARN 2025-09-10 21:58:29,212 [NioProcessor-2]-service.IoProcessor: Create a new selector. Selected is 0, delta = 0
WARN 2025-09-10 21:58:29,220 [pool-3-thread-2]-server.DsrvServerIoHandler: null
java.nio.channels.ClosedSelectorException: null
at sun.nio.ch.EPollSelectorImpl.ensureOpen(EPollSelectorImpl.java:98) ~[?:?]
at sun.nio.ch.EPollSelectorImpl.setEventOps(EPollSelectorImpl.java:243) ~[?:?]
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:101) ~[?:?]
at org.apache.mina.transport.socket.nio.NioProcessor.setInterestedInWrite(NioProcessor.java:371) ~[mina-core-2.0.27.jar:?]
at org.apache.mina.transport.socket.nio.NioProcessor.setInterestedInWrite(NioProcessor.java:47) ~[mina-core-2.0.27.jar:?]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.updateTrafficControl(AbstractPollingIoProcessor.java:585) [mina-core-2.0.27.jar:?]
at org.apache.mina.core.polling.AbstractPollingIoProcessor.updateTrafficControl(AbstractPollingIoProcessor.java:68) [mina-core-2.0.27.jar:?]
at org.apache.mina.core.service.SimpleIoProcessorPool.updateTrafficControl(SimpleIoProcessorPool.java:294) [mina-core-2.0.27.jar:?]
at org.apache.mina.core.service.SimpleIoProcessorPool.updateTrafficControl(SimpleIoProcessorPool.java:80) [mina-core-2.0.27.jar:?]
at org.apache.mina.core.session.AbstractIoSession.resumeRead(AbstractIoSession.java:748) [mina-core-2.0.27.jar:?]
at j4sf.connect.dsrv.endpoint.IoSessionEndPoint.resumeRead(IoSessionEndPoint.java:73) [DataServer-11.0.2.jar:?]
at j4sf.connect.dsrv.server.SerialExecutor.scheduleNext(SerialExecutor.java:94) [DataServer-11.0.2.jar:?]
at j4sf.connect.dsrv.server.SerialExecutor$LocalTask.run(SerialExecutor.java:140) [DataServer-11.0.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
INFO 2025-09-10 21:58:29,235 [NioProcessor-2]-server.DsrvServerIoHandler: Session Closed
Our code in the j4sf.connect.dsrv.server.SerialExecutor and j4sf.connect.dsrv.endpoint.IoSessionEndpoint is handling reading incoming messages on the IoSession. There is a message processing queues with a high and low watermark. At some point we have exceeded the high watermark and called suspendRead() on the IoSession. The ClosedConnectorException above has thrown when we called ioSession.resumeRead().
Our DsrvServerIoHandler extends org.apache.mina.core.service.IoHandlerAdapter - the exception is caught here where we override IoHandlerAdapter.exceptionCaught(IoSession, Throwable). At this point we log the exception and close the session, resulting in the final log message above after the session has closed.
We've been unable to reproduce this on a hardware/software platform under our control.
I don't see any mention of ClosedSelectorException in your mail archive since handling of them was fixed in DIRMINA-978<https://urldefense.com/v3/__https://issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$>>
Questions:
1. Could this be a race condition with the resumeRead causing the NioProcessor retrieving the old selection key just before it was closed, then attempting the read after it has been closed?
Yes, because the thread that deals with the selector replacement is
different from the thread processing the messages.
1. The IoSession is already suspended when this happens - is there any reason inherent in MINA why we shouldn't ignore this exception, at least once, and simply attempt to resumeRead() again. If I could reproduce the issue outside prod, I would of course try this myself.
If not, do you have any other advice on how the issue might be worked around.
I *think* (but this is from the top of my head) that the necessary step
to fix the issue is re-register the channel on the new selector, but it
has to be present. So as soon as the previous identified race condition
is fixed, this issue should bot occur. It would probably be enough to
add a lock section where we span a new selector that blocks any
operation on the selector. May be an AtomicReference to the selector
itself could help?
1. Might upgrade to MINA 2.1 or 2.2 help? I see nothing obvious I the release notes that suggests it would. We have already upgraded from 2.0.21 to 2.0.27 which seems to have reduced the frequency of the issue - perhaps due to the inclusion on DIRMINA-1169<https://urldefense.com/v3/__https://issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$>> in 2.0.24.
No, I'm afraid that won't help. Although I encourage you to migrate,
2.0.X is pretty old... (6 years since 2.1 has been release)
1. Finally, do you know anything about likely causes of the initial Epoll race condition the MINA code is working around. If it's caused by java, is there an OpenJDK issue open related to it? I couldn't find anything that seem to match. FWIW the system the issue occurs on is running RHEL 9.5 and the JDK is also Red Hat's java 11: Red_Hat-11.0.17.0.8-2.el7openjdkportable
As I mentionned above, JDK bug. MINA is not the only NIO framework
handling a spinning selector this way :/
Thanks!
Adam
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@mina.apache.org<ma...@mina.apache.org>
For additional commands, e-mail: users-help@mina.apache.org<ma...@mina.apache.org>
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
Re: Possible second race condition after EPoll spinning causing ClosedSelectorException
Posted by Emmanuel Lecharny <el...@gmail.com>.
Hi Adam,
I think I can cut a 2.0 and 2.1 release with the fix, yes. It will just
take a bit of time (like 2 or 3 weeks), I'm pretty busy atm before
getting some well deserved vacations next week.
On 12/16/25 15:18, Herring, Adam wrote:
> Hi Emmanuel,
>
> Thanks for putting in a fix for this in 2.2.5. We are migrating the head version of our software onto that.
>
> We’ve also implemented a workaround on some of our older code lines that are still live, do you think this fix would eventually be applied to 2.0.X though or is that not likely to happen?
>
> Thanks again,
> Adam
>
> From: Emmanuel Lecharny <el...@gmail.com>
> Sent: 22 October 2025 09:00
> To: Herring, Adam <Ad...@broadridge.com>; users@mina.apache.org
> Subject: Re: Possible second race condition after EPoll spinning causing ClosedSelectorException
>
> Hi Adam, On 9/25/25 17: 53, Herring, Adam wrote: Hi Emmanuel, Thanks for your helpful response and the link to the original JDK ticket. I guess it was closed without being truly fixed! Re-registering the channel on the new selector is exactly
> ZjQcmQRYFpfptBannerStart
> This Message Is From an Unknown Sender
> You have not previously corresponded with this sender. Use caution when clicking links or opening attachments. Report suspicious emails by clicking the "Report" button in your Outlook toolbar or forward to SIRT@Broadridge.com<ma...@Broadridge.com>.
> ZjQcmQRYFpfptBannerEnd
>
> Hi Adam,
> On 9/25/25 17:53, Herring, Adam wrote:
>
> Hi Emmanuel,
>
>
>
> Thanks for your helpful response and the link to the original JDK ticket. I guess it was closed without being truly fixed!
>
>
>
> Re-registering the channel on the new selector is exactly what NioProcessor.registerNewSelector() looks to be doing.
>
> Indeed. The idea is to loop on all the SelectionKey instances, and register each one of them on the new selector.
>
>
>
> Looking at this code in the mina sources again while reading your response, I noticed that this method is taking securing selectorLock.writeLock().lock() before it creates and registers the new selector (and releasing it at the end).
>
> yes, except for the allSessionsCount() method which *should* be protected:
>
> protected int allSessionsCount()
>
> {
>
> return selector.keys().size();
>
> }
>
>
>
> should be:
>
>
>
> protected int allSessionsCount()
>
> {
>
> selectorLock.readLock().lock();
>
>
>
>
>
> try {
>
> return selector.keys().size();
>
> } finally {
>
> selectorLock.readLock().unlock();
>
> }
>
> }
>
>
>
>
>
>
>
>
>
> NioProcessor.setInterestedInWrite(…) though is not getting the lock before it called key.interestOps(newInterestOps) which is where the ClosedSelectorException occurs.
>
> Looking at the SelectionKeyImpl.java code, here is what I see:
>
>
>
> we call:
>
> @Override
>
> protected void setInterestedInWrite(NioSession session, boolean isInterested) throws Exception {
>
> SelectionKey key = session.getSelectionKey();
>
>
>
> ...
>
>
>
> key.interestOps(newInterestOps);
>
> }
>
>
> which translates to:
>
> public SelectionKey interestOps(int ops) {
> ensureValid();
> return nioInterestOps(ops); <------
> }
>
> which calls:
>
>
>
> public SelectionKey nioInterestOps(int ops) {
> if ((ops & ~channel().validOps()) != 0)
> throw new IllegalArgumentException();
> channel.translateAndSetInterestOps(ops, this); <---------
> interestOps = ops;
> return this;
> }
>
> which itself calls:
>
>
>
> public void translateAndSetInterestOps(int ops, SelectionKeyImpl sk) {
> int newOps = 0;
> if ((ops & SelectionKey.OP_READ) != 0)
> newOps |= PollArrayWrapper.POLLIN;
> if ((ops & SelectionKey.OP_WRITE) != 0)
> newOps |= PollArrayWrapper.POLLOUT;
> if ((ops & SelectionKey.OP_CONNECT) != 0)
> newOps |= PollArrayWrapper.POLLCONN;
> sk.selector.putEventOps(sk, newOps);
>
>
>
> The last line leverage the selector, so yes, we most certainly need to protect the selector inside setInterestedInWrite and setInterestedInRead.
>
>
>
>
>
> Is this failure to get the lock here the issue?
>
>
>
> Thanks,
>
> Adam
>
>
>
> From: Emmanuel Lecharny <el...@gmail.com>
>
> Sent: 23 September 2025 13:29
>
> To: Herring, Adam <Ad...@broadridge.com.INVALID>; users@mina.apache.org<ma...@mina.apache.org>
>
> Subject: Re: Possible second race condition after EPoll spinning causing ClosedSelectorException
>
>
>
> Hi Adam, comments inline. . . On 9/23/25 10: 46, Herring, Adam wrote: > Hi Apache MINA team, > > I'm seeking advice on how to handle an apparent race condition encountered using mina 2. 0. 27, whether my analysis is correct and if this is
>
> ZjQcmQRYFpfptBannerStart
>
> This Message Is From an Unknown Sender
>
> You have not previously corresponded with this sender. Use caution when clicking links or opening attachments. Report suspicious emails by clicking the "Report" button in your Outlook toolbar or forward to SIRT@Broadridge.com<ma...@Broadridge.com>.
>
> ZjQcmQRYFpfptBannerEnd
>
>
>
> Hi Adam,
>
>
>
>
>
>
>
>
>
>
>
> comments inline...
>
>
>
>
>
>
>
> On 9/23/25 10:46, Herring, Adam wrote:
>
>
>
> Hi Apache MINA team,
>
> I'm seeking advice on how to handle an apparent race condition encountered using mina 2.0.27, whether my analysis is correct and if this is a known issue (I haven't identified one open in https://urldefense.com/v3/__https://issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$<https://urldefense.com/v3/__https:/issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/projects/DIRMINA/issues/DIRMINA-1186?filter=allopenissues__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBdYz_3sI$>).
>
> Issue Description
>
> Our application server uses MINA for NIO between a data cache and several client applications. Recently we have seen increased incidence of the following warning-level message in the data cache logs:
>
> Create a new selector. Selected is 0, delta = 0
>
> I found this is logged from the AbstractPollingIoProcessor and is accompanied by a comment block explaining that there is an Epoll race condition that can cause file descriptors not to be considered available and the selector must be closed and a new one registered to prevent 100% CPU use. It's not entirely clear whether this issue is in MINA, the JDK or the operating system, but either way this all seems fine and most of our application instances do not encounter a further problem.
>
>
>
> It's a JDK problem, I do'nt think it was ever fixed:
>
>
>
> https://urldefense.com/v3/__https://bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$<https://urldefense.com/v3/__https:/bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$><https://urldefense.com/v3/__https:/bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$><https://urldefense.com/v3/__https:/bugs.openjdk.org/browse/JDK-8011538__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBXBjAtn4$>
>
>
>
>
>
>
>
>
>
>
>
> On one production instance of our application it looks like a second race condition occurs when the selector is closed and replaced. The greater the frequency of the "Create a new selector. Selected is 0, delta = 0" message, the more likely that we encounter a ClosedSelectorException on another thread. Full stack trace and logs before and after:
>
> WARN 2025-09-10 21:58:29,212 [NioProcessor-2]-service.IoProcessor: Create a new selector. Selected is 0, delta = 0
>
> WARN 2025-09-10 21:58:29,220 [pool-3-thread-2]-server.DsrvServerIoHandler: null
>
> java.nio.channels.ClosedSelectorException: null
>
> at sun.nio.ch.EPollSelectorImpl.ensureOpen(EPollSelectorImpl.java:98) ~[?:?]
>
> at sun.nio.ch.EPollSelectorImpl.setEventOps(EPollSelectorImpl.java:243) ~[?:?]
>
> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:101) ~[?:?]
>
> at org.apache.mina.transport.socket.nio.NioProcessor.setInterestedInWrite(NioProcessor.java:371) ~[mina-core-2.0.27.jar:?]
>
> at org.apache.mina.transport.socket.nio.NioProcessor.setInterestedInWrite(NioProcessor.java:47) ~[mina-core-2.0.27.jar:?]
>
> at org.apache.mina.core.polling.AbstractPollingIoProcessor.updateTrafficControl(AbstractPollingIoProcessor.java:585) [mina-core-2.0.27.jar:?]
>
> at org.apache.mina.core.polling.AbstractPollingIoProcessor.updateTrafficControl(AbstractPollingIoProcessor.java:68) [mina-core-2.0.27.jar:?]
>
> at org.apache.mina.core.service.SimpleIoProcessorPool.updateTrafficControl(SimpleIoProcessorPool.java:294) [mina-core-2.0.27.jar:?]
>
> at org.apache.mina.core.service.SimpleIoProcessorPool.updateTrafficControl(SimpleIoProcessorPool.java:80) [mina-core-2.0.27.jar:?]
>
> at org.apache.mina.core.session.AbstractIoSession.resumeRead(AbstractIoSession.java:748) [mina-core-2.0.27.jar:?]
>
> at j4sf.connect.dsrv.endpoint.IoSessionEndPoint.resumeRead(IoSessionEndPoint.java:73) [DataServer-11.0.2.jar:?]
>
> at j4sf.connect.dsrv.server.SerialExecutor.scheduleNext(SerialExecutor.java:94) [DataServer-11.0.2.jar:?]
>
> at j4sf.connect.dsrv.server.SerialExecutor$LocalTask.run(SerialExecutor.java:140) [DataServer-11.0.2.jar:?]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
>
> at java.lang.Thread.run(Thread.java:829) [?:?]
>
> INFO 2025-09-10 21:58:29,235 [NioProcessor-2]-server.DsrvServerIoHandler: Session Closed
>
> Our code in the j4sf.connect.dsrv.server.SerialExecutor and j4sf.connect.dsrv.endpoint.IoSessionEndpoint is handling reading incoming messages on the IoSession. There is a message processing queues with a high and low watermark. At some point we have exceeded the high watermark and called suspendRead() on the IoSession. The ClosedConnectorException above has thrown when we called ioSession.resumeRead().
>
> Our DsrvServerIoHandler extends org.apache.mina.core.service.IoHandlerAdapter - the exception is caught here where we override IoHandlerAdapter.exceptionCaught(IoSession, Throwable). At this point we log the exception and close the session, resulting in the final log message above after the session has closed.
>
> We've been unable to reproduce this on a hardware/software platform under our control.
>
> I don't see any mention of ClosedSelectorException in your mail archive since handling of them was fixed in DIRMINA-978<https://urldefense.com/v3/__https://issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-978__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBMS-GwrI$>>
>
> Questions:
>
> 1. Could this be a race condition with the resumeRead causing the NioProcessor retrieving the old selection key just before it was closed, then attempting the read after it has been closed?
>
>
>
>
>
>
>
> Yes, because the thread that deals with the selector replacement is
>
>
>
> different from the thread processing the messages.
>
>
>
>
>
>
>
>
>
>
>
> 1. The IoSession is already suspended when this happens - is there any reason inherent in MINA why we shouldn't ignore this exception, at least once, and simply attempt to resumeRead() again. If I could reproduce the issue outside prod, I would of course try this myself.
>
> If not, do you have any other advice on how the issue might be worked around.
>
>
>
>
>
>
>
> I *think* (but this is from the top of my head) that the necessary step
>
>
>
> to fix the issue is re-register the channel on the new selector, but it
>
>
>
> has to be present. So as soon as the previous identified race condition
>
>
>
> is fixed, this issue should bot occur. It would probably be enough to
>
>
>
> add a lock section where we span a new selector that blocks any
>
>
>
> operation on the selector. May be an AtomicReference to the selector
>
>
>
> itself could help?
>
>
>
>
>
>
>
>
>
>
>
> 1. Might upgrade to MINA 2.1 or 2.2 help? I see nothing obvious I the release notes that suggests it would. We have already upgraded from 2.0.21 to 2.0.27 which seems to have reduced the frequency of the issue - perhaps due to the inclusion on DIRMINA-1169<https://urldefense.com/v3/__https://issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$><https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/DIRMINA-1169__;!!Ebr-cpPeAnfNniQ8HSAI-g_K5b7VKg!NbCVM_Tb4VZ_vqj39z3rMyVYgqSJttqNJGeI3p-J0iuMY3V7NzTMKrPAO5XcNB--SxJXHWIzsX5x2UMBw0EGOts$>> in 2.0.24.
>
>
>
> No, I'm afraid that won't help. Although I encourage you to migrate,
>
>
>
> 2.0.X is pretty old... (6 years since 2.1 has been release)
>
>
>
>
>
>
>
>
>
>
>
> 1. Finally, do you know anything about likely causes of the initial Epoll race condition the MINA code is working around. If it's caused by java, is there an OpenJDK issue open related to it? I couldn't find anything that seem to match. FWIW the system the issue occurs on is running RHEL 9.5 and the JDK is also Red Hat's java 11: Red_Hat-11.0.17.0.8-2.el7openjdkportable
>
>
>
> As I mentionned above, JDK bug. MINA is not the only NIO framework
>
>
>
> handling a spinning selector this way :/
>
>
>
>
>
>
>
>
>
>
>
> Thanks!
>
> Adam
>
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>
>
>
> ---------------------------------------------------------------------
>
>
>
> To unsubscribe, e-mail: users-unsubscribe@mina.apache.org<ma...@mina.apache.org>
>
>
>
> For additional commands, e-mail: users-help@mina.apache.org<ma...@mina.apache.org>
>
>
>
>
>
>
>
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.