You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Matthew Tyson <ma...@gmail.com> on 2011/11/15 18:00:38 UTC
Errors with NIO processor
Hey Guys,
We are seeing the following errors (in production of course, testing didn't
reveal this) after switching to NIO protocol.
This is Tomcat 7.0.22 on CentOS 6. There is a load balancer sending only
comet traffic to port 8080, where the NIO protocol is used.
Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.NullPointerException
Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint processSocket
SEVERE: Error allocating socket processor
java.lang.NullPointerException
at
org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
at java.lang.Thread.run(Thread.java:662)
Nov 15, 2011 8:39:52 AM
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
SEVERE: null
java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
for a request with Async state [STARTED]
at
org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
at
org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Exception in declaration()
I see more of the "Calling [asyncPostProcess()] is not valid for a request
with Async state [STARTED]" error by itself also. Here is the connector
setup:
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="20000"
redirectPort="8443" />
Any direction on where to look for the cause?
Thanks,
Matt
Re: Errors with NIO processor
Posted by Matthew Tyson <ma...@gmail.com>.
Thanks Bob. It doesn't seem to be a load problem. It happens consistently
even for just 1 user.
If I switch the connector back to HTTP/1.1, instead of NIO, the problem
goes away.
Sometimes, there doesn't appear to be an error in catalina.out, but there
is a response with no body, just headers like this:
Date Wed, 16 Nov 2011 00:43:58 GMT
Server Apache-Coyote/1.1
Content-Type text/html;charset=ISO-8859-1
Cache-Control max-age=2
Expires Wed, 16 Nov 2011 00:44:00 GMT
Set-Cookie xgh=gnweb10; path=/; BIGipCookie=000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000
Vary User-Agent,Accept-Encoding
P3P policyref=http://www.company.net/w3c/p3p.xml<http://www.gaggle.net/w3c/p3p.xml>,
CP=ALL
Content-Encoding gzip
Content-Length 20
Connection close
On Tue, Nov 15, 2011 at 4:51 PM, Bob Hall <rf...@yahoo.com> wrote:
> Matt,
>
> Did the testing include load testing? Have you checked the "open file
> limit" values?
>
> If not, you may be running into an "open file limit" for the OS and/or
> user that is running Tomcat.
>
> - Bob
>
>
> ________________________________
> From: Matthew Tyson <ma...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Tuesday, November 15, 2011 4:18 PM
> Subject: Re: Errors with NIO processor
>
> Is there more info I can provide to help diagnose this error? It is
> killing us.
>
> Thanks,
>
> Matt
>
>
>
> On Tue, Nov 15, 2011 at 9:00 AM, Matthew Tyson
> <ma...@gmail.com>wrote:
>
> > Hey Guys,
> >
> > We are seeing the following errors (in production of course, testing
> > didn't reveal this) after switching to NIO protocol.
> >
> > This is Tomcat 7.0.22 on CentOS 6. There is a load balancer sending only
> > comet traffic to port 8080, where the NIO protocol is used.
> >
> > Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint
> > processSocket
> > SEVERE: Error allocating socket processor
> > java.lang.NullPointerException
> > Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint
> > processSocket
> > SEVERE: Error allocating socket processor
> > java.lang.NullPointerException
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> > at
> > org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> > at java.lang.Thread.run(Thread.java:662)
> > Nov 15, 2011 8:39:52 AM
> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> > SEVERE: null
> > java.lang.IllegalStateException: Calling [asyncPostProcess()] is not
> valid
> > for a request with Async state [STARTED]
> > at
> >
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> > at
> >
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> > at
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> > at
> >
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Exception in declaration()
> >
> > I see more of the "Calling [asyncPostProcess()] is not valid for a
> request
> > with Async state [STARTED]" error by itself also. Here is the connector
> > setup:
> >
> > <Connector port="8080"
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > connectionTimeout="20000"
> > redirectPort="8443" />
> >
> > Any direction on where to look for the cause?
> >
> > Thanks,
> >
> > Matt
> >
>
Re: Errors with NIO processor
Posted by Bob Hall <rf...@yahoo.com>.
Matt,
Did the testing include load testing? Have you checked the "open file limit" values?
If not, you may be running into an "open file limit" for the OS and/or user that is running Tomcat.
- Bob
________________________________
From: Matthew Tyson <ma...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Tuesday, November 15, 2011 4:18 PM
Subject: Re: Errors with NIO processor
Is there more info I can provide to help diagnose this error? It is
killing us.
Thanks,
Matt
On Tue, Nov 15, 2011 at 9:00 AM, Matthew Tyson
<ma...@gmail.com>wrote:
> Hey Guys,
>
> We are seeing the following errors (in production of course, testing
> didn't reveal this) after switching to NIO protocol.
>
> This is Tomcat 7.0.22 on CentOS 6. There is a load balancer sending only
> comet traffic to port 8080, where the NIO protocol is used.
>
> Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> at
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> at java.lang.Thread.run(Thread.java:662)
> Nov 15, 2011 8:39:52 AM
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> SEVERE: null
> java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
> for a request with Async state [STARTED]
> at
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> at
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in declaration()
>
> I see more of the "Calling [asyncPostProcess()] is not valid for a request
> with Async state [STARTED]" error by itself also. Here is the connector
> setup:
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="20000"
> redirectPort="8443" />
>
> Any direction on where to look for the cause?
>
> Thanks,
>
> Matt
>
Re: Errors with NIO processor
Posted by Matthew Tyson <ma...@gmail.com>.
Is there more info I can provide to help diagnose this error? It is
killing us.
Thanks,
Matt
On Tue, Nov 15, 2011 at 9:00 AM, Matthew Tyson
<ma...@gmail.com>wrote:
> Hey Guys,
>
> We are seeing the following errors (in production of course, testing
> didn't reveal this) after switching to NIO protocol.
>
> This is Tomcat 7.0.22 on CentOS 6. There is a load balancer sending only
> comet traffic to port 8080, where the NIO protocol is used.
>
> Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint
> processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> at
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> at java.lang.Thread.run(Thread.java:662)
> Nov 15, 2011 8:39:52 AM
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> SEVERE: null
> java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
> for a request with Async state [STARTED]
> at
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> at
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in declaration()
>
> I see more of the "Calling [asyncPostProcess()] is not valid for a request
> with Async state [STARTED]" error by itself also. Here is the connector
> setup:
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="20000"
> redirectPort="8443" />
>
> Any direction on where to look for the cause?
>
> Thanks,
>
> Matt
>
Re: Errors with NIO processor
Posted by Mark Thomas <ma...@apache.org>.
On 15/11/2011 17:00, Matthew Tyson wrote:
> Hey Guys,
>
> We are seeing the following errors (in production of course, testing didn't
> reveal this) after switching to NIO protocol.
>
> This is Tomcat 7.0.22 on CentOS 6. There is a load balancer sending only
> comet traffic to port 8080, where the NIO protocol is used.
>
> Nov 15, 2011 8:39:29 AM org.apache.tomcat.util.net.NioEndpoint processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> Nov 15, 2011 8:39:51 AM org.apache.tomcat.util.net.NioEndpoint processSocket
> SEVERE: Error allocating socket processor
> java.lang.NullPointerException
> at
> org.apache.tomcat.util.net.NioEndpoint.processSocket(NioEndpoint.java:712)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:1200)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1136)
> at java.lang.Thread.run(Thread.java:662)
> Nov 15, 2011 8:39:52 AM
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler process
> SEVERE: null
> java.lang.IllegalStateException: Calling [asyncPostProcess()] is not valid
> for a request with Async state [STARTED]
> at
> org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:202)
> at
> org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:104)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:519)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1550)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in declaration()
>
> I see more of the "Calling [asyncPostProcess()] is not valid for a request
> with Async state [STARTED]" error by itself also. Here is the connector
> setup:
>
> <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="20000"
> redirectPort="8443" />
>
> Any direction on where to look for the cause?
It could be a bug somewhere in the NIO connector. There has been a lot
of refactoring to reduce duplication between the connectors. In the long
term that should reduce the bugs and makes those that remain easier to
fix. In the short term, there have been a couple of regressions.
What we really need is a reproducible test case. The simpler, the better.
It could also be an application bug. A test case would help identify
that too.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org