You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Antoine Bonavita <an...@stickyads.tv> on 2015/07/22 16:54:22 UTC
NullPointerException in NioEndpoint under load (but not only)
Hello,
Before I start with my question and stack trace, let me give you a bit
of context:
* Tomcat 7.0.61
* OpenJDK1.7.0_79
* Debian7
* Tomcat is running "behind" an nginx server (reverse proxying).
* As we have pretty stringent constraints on response time, nginx is
configured to reply to the browser after 2s if it has not heard back
from Tomcat.
* As implied in the email subject, we are using the http-nio connector.
This is because our code is calling many webservices in parallel and we
don't want to be stuck waiting for one of them (after 300ms we timeout
the request and consider they will not answer).
Now, my problem is that once in a while (with a higher probability under
load), I get this exception:
java.lang.NullPointerException
at
org.apache.tomcat.util.net.NioEndpoint.processSocketAsync(NioEndpoint.java:730)
~[tomcat-coyote.jar:7.0.61]
at
org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84)
~[tomcat-coyote.jar:7.0.61]
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:764)
~[tomcat-coyote.jar:7.0.61]
at org.apache.coyote.Response.action(Response.java:174)
~[tomcat-coyote.jar:7.0.61]
at org.apache.coyote.Response.finish(Response.java:274)
~[tomcat-coyote.jar:7.0.61]
at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
~[catalina.jar:7.0.61]
at
org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112)
~[catalina.jar:7.0.61]
at
networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:239)
~[HttpCommand.class:?]
at
com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:323)
~[AuctioneerResponseWriter.class:?]
at
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.bidder.marketplace.MarketPlaceBidSerializationWriter.respondToClient(MarketPlaceBidSerializationWriter.java:92)
~[MarketPlaceBidSerializationWriter.class:?]
at
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24)
~[BidSerializationListener.class:?]
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
~[httpcore-4.4.1.jar:4.4.1]
at
com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:166)
~[OpenRTBBid.class:?]
at
com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:111)
~[AdmReceived.class:?]
at
com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:32)
~[AdmReceived.class:?]
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
~[httpcore-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
~[httpasyncclient-4.1.jar:4.1]
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:412)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:305)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
~[httpasyncclient-4.1.jar:4.1]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
~[httpasyncclient-4.1.jar:4.1]
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:116)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
~[httpcore-nio-4.4.1.jar:4.4.1]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
I looked a bit at the NioEndpoint and AbstractProcessor code and it
looks to me like the only possible cause is that
AbstractProcessor.socketWrapper is null at this point. Can someone
confirm my analysis ?
Now, I tried to figure out how this can happen and that's where I got
lost in the code. So, if someone could help me figure out what in our
code could lead to this, that would be very much appreciated.
Last, but not least, there seems to be a correlation between those
exceptions and spikes with the LimitLatch that never go down. My
question is : could this NPE result in the LimitLatch not going down ?
Thanks for your help and obviously, do not hesitate to ask me questions
if you need more information I did not provide.
A.
--
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: NullPointerException in NioEndpoint under load (but not only)
Posted by Antoine Bonavita <an...@stickyads.tv>.
Hello,
Never had any feedback on this, but I think I'm running into this bug:
https://bz.apache.org/bugzilla/show_bug.cgi?id=57943
At least, setting the socket.bufferPool to 0 tremendously helped and I
am not seeing this excpetion anymore.
Waiting for 7.0.64 release to upgrade but I have good hope.
A.
On 07/22/2015 04:54 PM, Antoine Bonavita wrote:
> Hello,
>
> Before I start with my question and stack trace, let me give you a bit
> of context:
> * Tomcat 7.0.61
> * OpenJDK1.7.0_79
> * Debian7
> * Tomcat is running "behind" an nginx server (reverse proxying).
> * As we have pretty stringent constraints on response time, nginx is
> configured to reply to the browser after 2s if it has not heard back
> from Tomcat.
> * As implied in the email subject, we are using the http-nio connector.
> This is because our code is calling many webservices in parallel and we
> don't want to be stuck waiting for one of them (after 300ms we timeout
> the request and consider they will not answer).
>
> Now, my problem is that once in a while (with a higher probability under
> load), I get this exception:
> java.lang.NullPointerException
> at
> org.apache.tomcat.util.net.NioEndpoint.processSocketAsync(NioEndpoint.java:730)
> ~[tomcat-coyote.jar:7.0.61]
> at
> org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84)
> ~[tomcat-coyote.jar:7.0.61]
> at
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:764)
> ~[tomcat-coyote.jar:7.0.61]
> at org.apache.coyote.Response.action(Response.java:174)
> ~[tomcat-coyote.jar:7.0.61]
> at org.apache.coyote.Response.finish(Response.java:274)
> ~[tomcat-coyote.jar:7.0.61]
> at
> org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
> ~[catalina.jar:7.0.61]
> at
> org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112)
> ~[catalina.jar:7.0.61]
> at
> networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:239)
> ~[HttpCommand.class:?]
> at
> com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:323)
> ~[AuctioneerResponseWriter.class:?]
> at
> com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
> ~[BidSerializationListener.class:?]
> at
> com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
> ~[BidSerializationListener.class:?]
> at
> com.stickyadstv.adex.bidder.marketplace.MarketPlaceBidSerializationWriter.respondToClient(MarketPlaceBidSerializationWriter.java:92)
> ~[MarketPlaceBidSerializationWriter.class:?]
> at
> com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
> ~[BidSerializationListener.class:?]
> at
> com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
> ~[BidSerializationListener.class:?]
> at
> com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24)
> ~[BidSerializationListener.class:?]
> at
> org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
> ~[httpcore-4.4.1.jar:4.4.1]
> at
> com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:166)
> ~[OpenRTBBid.class:?]
> at
> com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:111)
> ~[AdmReceived.class:?]
> at
> com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:32)
> ~[AdmReceived.class:?]
> at
> org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
> ~[httpcore-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
> ~[httpasyncclient-4.1.jar:4.1]
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:412)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:305)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
> ~[httpasyncclient-4.1.jar:4.1]
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
> ~[httpasyncclient-4.1.jar:4.1]
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:116)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
> at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
>
> I looked a bit at the NioEndpoint and AbstractProcessor code and it
> looks to me like the only possible cause is that
> AbstractProcessor.socketWrapper is null at this point. Can someone
> confirm my analysis ?
>
> Now, I tried to figure out how this can happen and that's where I got
> lost in the code. So, if someone could help me figure out what in our
> code could lead to this, that would be very much appreciated.
>
> Last, but not least, there seems to be a correlation between those
> exceptions and spikes with the LimitLatch that never go down. My
> question is : could this NPE result in the LimitLatch not going down ?
>
> Thanks for your help and obviously, do not hesitate to ask me questions
> if you need more information I did not provide.
>
> A.
>
--
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org