Re: NullPointerException in NioEndpoint under load (but not only)

[Antoine Bonavita <an...@stickyads.tv>]

Hello,

Never had any feedback on this, but I think I'm running into this bug:
https://bz.apache.org/bugzilla/show_bug.cgi?id=57943

At least, setting the socket.bufferPool to 0 tremendously helped and I 
am not seeing this excpetion anymore.

Waiting for 7.0.64 release to upgrade but I have good hope.

A.

On 07/22/2015 04:54 PM, Antoine Bonavita wrote:
> Hello,
>
> Before I start with my question and stack trace, let me give you a bit
> of context:
> * Tomcat 7.0.61
> * OpenJDK1.7.0_79
> * Debian7
> * Tomcat is running "behind" an nginx server (reverse proxying).
> * As we have pretty stringent constraints on response time, nginx is
> configured to reply to the browser after 2s if it has not heard back
> from Tomcat.
> * As implied in the email subject, we are using the http-nio connector.
> This is because our code is calling many webservices in parallel and we
> don't want to be stuck waiting for one of them (after 300ms we timeout
> the request and consider they will not answer).
>
> Now, my problem is that once in a while (with a higher probability under
> load), I get this exception:
> java.lang.NullPointerException
>          at
> org.apache.tomcat.util.net.NioEndpoint.processSocketAsync(NioEndpoint.java:730)
> ~[tomcat-coyote.jar:7.0.61]
>          at
> org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84)
> ~[tomcat-coyote.jar:7.0.61]
>          at
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:764)
> ~[tomcat-coyote.jar:7.0.61]
>          at org.apache.coyote.Response.action(Response.java:174)
> ~[tomcat-coyote.jar:7.0.61]
>          at org.apache.coyote.Response.finish(Response.java:274)
> ~[tomcat-coyote.jar:7.0.61]
>          at
> org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
> ~[catalina.jar:7.0.61]
>          at
> org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112)
> ~[catalina.jar:7.0.61]
>          at
> networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:239)
> ~[HttpCommand.class:?]
>          at
> com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:323)
> ~[AuctioneerResponseWriter.class:?]
>          at
> com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
> ~[BidSerializationListener.class:?]
>          at
> com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
> ~[BidSerializationListener.class:?]
>          at
> com.stickyadstv.adex.bidder.marketplace.MarketPlaceBidSerializationWriter.respondToClient(MarketPlaceBidSerializationWriter.java:92)
> ~[MarketPlaceBidSerializationWriter.class:?]
>          at
> com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
> ~[BidSerializationListener.class:?]
>          at
> com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
> ~[BidSerializationListener.class:?]
>          at
> com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24)
> ~[BidSerializationListener.class:?]
>          at
> org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
> ~[httpcore-4.4.1.jar:4.4.1]
>          at
> com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:166)
> ~[OpenRTBBid.class:?]
>          at
> com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:111)
> ~[AdmReceived.class:?]
>          at
> com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:32)
> ~[AdmReceived.class:?]
>          at
> org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
> ~[httpcore-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
> ~[httpasyncclient-4.1.jar:4.1]
>          at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:412)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:305)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
> ~[httpasyncclient-4.1.jar:4.1]
>          at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
> ~[httpasyncclient-4.1.jar:4.1]
>          at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:116)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
> ~[httpcore-nio-4.4.1.jar:4.4.1]
>          at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
>
> I looked a bit at the NioEndpoint and AbstractProcessor code and it
> looks to me like the only possible cause is that
> AbstractProcessor.socketWrapper is null at this point. Can someone
> confirm my analysis ?
>
> Now, I tried to figure out how this can happen and that's where I got
> lost in the code. So, if someone could help me figure out what in our
> code could lead to this, that would be very much appreciated.
>
> Last, but not least, there seems to be a correlation between those
> exceptions and spikes with the LimitLatch that never go down. My
> question is : could this NPE result in the LimitLatch not going down ?
>
> Thanks for your help and obviously, do not hesitate to ask me questions
> if you need more information I did not provide.
>
> A.
>

-- 
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org