You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by "Denis Rogov (JIRA)" <ji...@apache.org> on 2008/06/27 11:27:45 UTC
[jira] Created: (HTTPCORE-164) DefaultConnectingIOReactor throws
ConcurrentModificationException in processEvents() method
DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
-------------------------------------------------------------------------------------------
Key: HTTPCORE-164
URL: https://issues.apache.org/jira/browse/HTTPCORE-164
Project: HttpComponents HttpCore
Issue Type: Bug
Components: HttpCore NIO
Affects Versions: 4.0-beta2
Environment: Win XP
JDK 1.6.0-b105
httpcore-nio-beta2-20080510.140437-10.jar
httpcore-beta2-20080510.140437-10.jar
Reporter: Denis Rogov
Priority: Critical
Using httpcomponents nio to serve as http client.
Occasionally getting exception that stops reactor and therefore all engine work:
Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:827)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[jira] Commented: (HTTPCORE-164) DefaultConnectingIOReactor throws
ConcurrentModificationException in processEvents() method
Posted by "Denis Rogov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608775#action_12608775 ]
Denis Rogov commented on HTTPCORE-164:
--------------------------------------
Oleg, at this moment I still found no test case to reproduce this. The thing is that it is being reproduced while server is running for a few hours.
But now I have few updates on this problem. First of all, this is not a root cause.
Here is the chronology of the incident:
1. For some reasons not driven from application DefaultConnectingIOReactor#execute(IOEventDispatch eventDispatch); method exits without throwing IOException or any other exception. Code like this produced log output like this:
try {
ioReactor.execute(ioEventDispatch);
} catch (InterruptedIOException ex) {
getLog().info("IO reactor execution has been interrupted.");
} catch (IOException e) {
getLog().error("IO exception occured.", e);
}
getLog().info("IO reactor execution has been finished.");
10:44:34,648 INFO [nioEventListener] connectionOpen
10:44:34,651 INFO [httpManagerAsyncEngine] IO reactor execution has been finished.
10:44:34,656 INFO [nioEventListener] [closed] connectionClosed
After that we got few other exceptions from the httpcomponenets code, one to exception handler, next in the thread that started ioReactor.execute(ioEventDispatch); (named "IO Reactor Execution Thread")
10:44:34,669 ERROR [httpManagerAsyncEngine] RuntimeException occured while processing request
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at org.apache.http.impl.nio.reactor.IOSessionImpl.setEvent(IOSessionImpl.java:111)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.submitRequest(DefaultNHttpClientConnection.java:214)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.requestReady(AsyncNHttpClientHandler.java:183)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.connected(AsyncNHttpClientHandler.java:111)
at org.apache.http.nio.protocol.BufferingHttpClientHandler.connected(BufferingHttpClientHandler.java:94)
at org.apache.http.impl.nio.DefaultClientIOEventDispatch.connected(DefaultClientIOEventDispatch.java:87)
at org.apache.http.impl.nio.reactor.BaseIOReactor.keyCreated(BaseIOReactor.java:182)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:246)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:153)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:70)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:321)
at java.lang.Thread.run(Thread.java:619)
10:44:34,684 INFO [nioEventListener] [closed] connectionClosed
10:44:34,686 INFO [nioEventListener] [closed] connectionClosed
10:44:34,687 INFO [nioEventListener] [closed] connectionClosed
10:44:34,689 INFO [nioEventListener] [closed] connectionClosed
Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:827)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
at <>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:282)
at java.lang.Thread.run(Thread.java:619)
The other important piece of code for this issue is reactor initializer block:
ioReactor = new DefaultConnectingIOReactor(3, httpParams);
((DefaultConnectingIOReactor) ioReactor).setExceptionHandler(new IOReactorExceptionHandler() {
public boolean handle(IOException ex) {
getLog().error("IOException occured while processing request", ex);
// in any case we should preserve ioreactor alive
return true;
}
public boolean handle(RuntimeException ex) {
getLog().error("RuntimeException occured while processing request", ex);
// in any case we should preserve ioreactor alive
return true;
}
As a result, the worst thing is that we got the reactor thread dead, which actually blocked all further application work. Do you have ideas why it could happen? And the most important question is how do you suggest to catch the cases when reactor or one of its workers crashed? How to resurrect it to go on with serving requests? Once it became in state shut down SHUT_DOWN I invoked execute method once again, but seems it is not a right way because it didn't got the control and returned from the method w/o exception immdiately.
Regards,
Denis
});
> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
> -------------------------------------------------------------------------------------------
>
> Key: HTTPCORE-164
> URL: https://issues.apache.org/jira/browse/HTTPCORE-164
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.0-beta2
> Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
> Reporter: Denis Rogov
> Priority: Critical
>
> Using httpcomponents nio to serve as http client.
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[jira] Commented: (HTTPCORE-164) DefaultConnectingIOReactor throws
ConcurrentModificationException in processEvents() method
Posted by "Andreas Andreou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608719#action_12608719 ]
Andreas Andreou commented on HTTPCORE-164:
------------------------------------------
Hi Oleg,
we're trying to find what causes this behavior - hopefully, that'll help with the testcase.
For the record, i'm using (Win Server 2003 + debian 2.6.18-6-686):
jdk 1.6.0_10-beta-b23
httpcore-nio-4.0-beta2.jar
httpcore-4.0-beta2.jar
> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
> -------------------------------------------------------------------------------------------
>
> Key: HTTPCORE-164
> URL: https://issues.apache.org/jira/browse/HTTPCORE-164
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.0-beta2
> Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
> Reporter: Denis Rogov
> Priority: Critical
>
> Using httpcomponents nio to serve as http client.
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[jira] Commented: (HTTPCORE-164) DefaultConnectingIOReactor throws
ConcurrentModificationException in processEvents() method
Posted by "Oleg Kalnichevski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608718#action_12608718 ]
Oleg Kalnichevski commented on HTTPCORE-164:
--------------------------------------------
Denis,
I cannot see how the selection set could be modified concurrently, at least just by looking at the source code. I need more input on how the problem can be reproduced, preferably a test case.
Besides, please make sure you are using the latest update of the JDK 1.6 (update 10). This may as well be a JRE bug.
Oleg
> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
> -------------------------------------------------------------------------------------------
>
> Key: HTTPCORE-164
> URL: https://issues.apache.org/jira/browse/HTTPCORE-164
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.0-beta2
> Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
> Reporter: Denis Rogov
> Priority: Critical
>
> Using httpcomponents nio to serve as http client.
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[jira] Resolved: (HTTPCORE-164) DefaultConnectingIOReactor throws
ConcurrentModificationException in processEvents() method
Posted by "Oleg Kalnichevski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kalnichevski resolved HTTPCORE-164.
----------------------------------------
Resolution: Cannot Reproduce
Denis, Andreas, please re-open this issue if there are any new findings. Closing as CANNOT_REPRODUCE for now.
Oleg
> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
> -------------------------------------------------------------------------------------------
>
> Key: HTTPCORE-164
> URL: https://issues.apache.org/jira/browse/HTTPCORE-164
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.0-beta2
> Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
> Reporter: Denis Rogov
> Priority: Critical
>
> Using httpcomponents nio to serve as http client.
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
[jira] Commented: (HTTPCORE-164) DefaultConnectingIOReactor throws
ConcurrentModificationException in processEvents() method
Posted by "Oleg Kalnichevski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HTTPCORE-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608793#action_12608793 ]
Oleg Kalnichevski commented on HTTPCORE-164:
--------------------------------------------
> For some reasons not driven from application DefaultConnectingIOReactor#execute(IOEventDispatch eventDispatch); method exits
> without throwing IOException or any other exception.
Bizarre. As far as I can tell I/O reactor can terminate either due to an IOReactorException or an explicit shutdown() invocation.
Anyway, IO reactor is not supposed to terminate quietly if something is not okay. In any case it should attempt to cleanly shutdown all its I/O dispatch worker before calling it quits.
Very, very odd.
> public boolean handle(RuntimeException ex) {
> getLog().error("RuntimeException occured while processing request", ex);
> // in any case we should preserve ioreactor alive
> return true;
>
This pretty much explains it. I would VERY STRONGLY advise against doing that. It is much better to let the reactor terminate cleanly on an runtime exception and restart it instead of leaving it running in a zombie state.
What you are doing is very, very wrong.
> And the most important question is how do you suggest to catch the cases when reactor or one of its workers crashed?
First off, you should not ignore exceptions indiscriminately. You should allow fatal exceptions to propagate. This will enable the I/O reactor to react to an exception and shut down its I/O dispatch threads cleanly. If the reactor terminates due to an exception is should be safe just to restart it.
Oleg
> DefaultConnectingIOReactor throws ConcurrentModificationException in processEvents() method
> -------------------------------------------------------------------------------------------
>
> Key: HTTPCORE-164
> URL: https://issues.apache.org/jira/browse/HTTPCORE-164
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore NIO
> Affects Versions: 4.0-beta2
> Environment: Win XP
> JDK 1.6.0-b105
> httpcore-nio-beta2-20080510.140437-10.jar
> httpcore-beta2-20080510.140437-10.jar
> Reporter: Denis Rogov
> Priority: Critical
>
> Using httpcomponents nio to serve as http client.
> Occasionally getting exception that stops reactor and therefore all engine work:
> Exception in thread "IO Reactor Execution Thread" java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:827)
> at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:93)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:162)
> at <...>HttpManagerApacheAsyncEngineImpl$1.run(HttpManagerApacheAsyncEngineImpl.java:294)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org